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Related Application 

20 [0001] The subject matter of this application is related to the subject 

matter in a co-pending non-provisional application entitled, "Method and 
Apparatus for Manipulating Two-Dimensional Windows Within a Three- 
Dimensional Display Model," by inventor Hideya Kawahara having serial number 
10/663,609, and filing date 15 September 2003 (Attorney Docket No. SUN04- 

25 0195-EKL). 
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BACKGROUND 



Field of the Invention 

[0002] The present invention relates to user interfaces for computer 
5 systems. More specifically, the present invention relates to a method and an 
apparatus for using visual effects to convey a usage context of a computational 
resource. 

Related Art 

10 [0003] Today, most personal computers and other high-end devices 

support window-based graphical user interfaces (GUIs), which were originally 
developed back in the 1980's. These window-based interfaces allow a user to 
manipulate windows through a pointing device (such as a mouse), in much the 
same way that pages can be manipulated on a desktop. However, because of 

1 5 limitations on graphical processing power at the time windows were being 
developed, many of the design decisions for windows were made with 
computational efficiency in mind. In particular, window-based systems provide a 
very flat (two-dimensional) 2D user experience, and windows are typically 
manipulated using operations that keep modifications of display pixels to a 

20 minimum. Even today's desktop environments like Microsoft Windows 

(distributed by the Microsoft Corporation of Redmond, Washington) include 
vestiges of design decisions made back then. 

[0004] In recent years, because of increasing computational requirements 
of 3D applications, especially 3D games, the graphical processing power of 

25 personal computers and other high-end devices has increased dramatically. For 
example, a middle range PC graphics card, the "GeForce2 GTS" distributed by the 
NVIDIA Corporation of Sunnyvale, California, provides a 3D rendering speed of 
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25 million polygon-per-second, and Microsoft's "Xbox" game console provides 
125 million polygon-per- second. These numbers are significantly better than 
those of high-end graphics workstation in the early 1990's, which cost tens of 
thousands (and even hundreds of thousands) of dollars. 
5 [0005] As graphical processing power has increased in recent years, a 

number of 3D user interfaces have been developed. These 3D interfaces typically 
allow a user to navigate through and manipulate 3D objects. However, these 3D 
interfaces are mainly focused on exploiting 3D capabilities, while little attention 
has been given to supporting existing, legacy window-based 2D applications 
10 within these 3D user interfaces. 

[0006] Hence, what needed is a method and an apparatus for leveraging 
the capabilities of 3D interfaces to convey more information about 2D 
applications running in a 3D workspace. 

15 SUMMARY 

[0007] One embodiment of the present invention provides a system that 
facilitates using visual effects within a three-dimensional (3D) display 
environment to indicate a usage context of a computational resource, which is 
represented by a 3D object in the 3D display environment. During operation, the 

20 system determines the usage context of the computational resource. Next, the 
system determines visual effects corresponding to the usage context to apply to 
the 3D object which represents the computational resource. Finally, the system 
applies the visual effects to the 3D object. In this way, the usage context of the 
corresponding computational resource can be determined by viewing the 3D 

25 object within the 3D display environment. 

[0008] In a variation on this embodiment, the computational resource can 
include one of a file, a folder, and an application. 
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[0009] In a variation on this embodiment, the usage context of the 
computational resource can specify a time at which the computational resource 
was created, a last time the computational resource was accessed, a frequency of 
usage of the computational resource, an owner of the computational resource, a 
5 location of the computational resource within the 3D display environment, and a 
group membership of the computational resource. 

[0010] In a variation on this embodiment, the visual effects can include a 
location of the 3D object, an orientation of the 3D object, a color of the 3D object, 
a texture of the 3D object, a surface pattern on the 3D object, a bump map for the 
10 3D object, a size of the 3D object, lighting for the 3D object, transparency of the 
3D object, reflectiveness of the 3D object, and simulated degradation of the 3D 
object. 

[0011] In a variation on this embodiment, the system receives input from a 
user to alter visual effects for the 3D object, and then alters visual effects for the 
15 3D object in response to the input. 

[0012] In a variation on this embodiment, visual effects for the 3D object 
change over time, thereby indicating aging of the associated computational 
resource. 

[0013] In a variation on this embodiment, the visual effects are applied to 
20 a group of 3D objects whose associated computational resources have a similar 
usage context to the computational resource associated with the 3D object. 

[0014] In a further variation, applying the visual effects to the group of 3D 
objects involves highlighting the group of 3D objects by using a scoped spotlight 
within the 3D display environment. 
25 [0015] In a further variation, applying the visual effects to the group of 3D 

objects involves orienting the group of 3D objects such that the surfaces of the 3D 
objects have the same angle. 
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[0016] In a variation on this embodiment, applying the visual effects 
involves applying a view direction within the 3D display environment to make 
visible a subset of 3D objects in the 3D display environment which constitute a 
workspace. 

5 [0017] In a variation on this embodiment, applying the visual effects 

involves changing a lighting position and corresponding shadows cast by the 3D 
object to indicate time of day. 

[0018] In a variation on this embodiment, applying the visual effects 
involves changing a fog parameter within the 3D display environment to indicate 
10 morning fog. 

[0019] In a variation on this embodiment, applying the visual effects 
involves changing a background color of the 3D display environment, wherein the 
background color is reflected by the 3D object. 



1 5 BRIEF DESCRIPTION OF THE FIGURES 

[0020] FIG. 1 illustrates a 3D display model with supporting components 
in accordance with an embodiment of the present invention. 

[0021] FIG. 2 presents a flow chart illustrating how input from a pointing 
device is communicated to an application associated with a window in a 3D 
20 display model in accordance with an embodiment of the present invention. 

[0022] FIG. 3 presents a flow chart illustrating using visual effects to 
convey usage context of computational resources in accordance with an 
embodiment of the present invention. 

[0023] FIG. 4A illustrates a group of windows in accordance with an 
25 embodiment of the present invention. 
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[0024] FIG. 4B illustrates a group of windows with visual effects 
indicating groups of windows in accordance with an embodiment of the present 
invention. 

[0025] FIG. 4C presents a flow chart illustrating the process of using 
5 visual effects to convey usage context for a group of computational resources in 
accordance with an embodiment of the present invention. 

[0026] FIG. 5 illustrates a number of virtual workspaces in accordance 
with an embodiment of the present invention. 

[0027] FIG. 6 presents a flow chart illustrating the process of environment 
10 mapping in accordance with an embodiment of the present invention. 

DETAILED DESCRIPTION 
[0028] The following description is presented to enable any person skilled 
in the art to make and use the invention, and is provided in the context of a parti- 

15 cular application and its requirements. Various modifications to the disclosed 
embodiments will be readily apparent to those skilled in the art, and the general 
principles defined herein may be applied to other embodiments and applications 
without departing from the spirit and scope of the present invention. Thus, the 
present invention is not intended to be limited to the embodiments shown, but is 

20 to be accorded the widest scope consistent with the principles and features 
disclosed herein. 

[0029] The data structures and code described in this detailed description 
are typically stored on a computer readable storage medium, which may be any 
device or medium that can store code and/or data for use by a computer system. 
25 This includes, but is not limited to, magnetic and optical storage devices such as 
disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs 
or digital video discs), and computer instruction signals embodied in a 
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transmission medium (with or without a carrier wave upon which the signals are 
modulated). For example, the transmission medium may include a 
communications network, such as the Internet. 



5 3D Display Model 

[0030] FIG. 1 illustrates 3D display model 102 with supporting 
components in accordance with an embodiment of the present invention. More 
specifically, the top portion of FIG. 1 illustrates 3D display model 102, which 
includes light source 103 and a number of 3D objects including window 108, 

1 0 window 1 1 0, and 3D object 111. Note that windows 1 08 and 1 1 0 are actually 3D 
objects which represent 2D windows. Hence, windows 108 and 1 10 can be 
moved and rotated within 3D display model 102, while they provide a 2D output 
and receive input for associated 2D applications. 3D display model 102 can 
additionally include a background (which is not shown). 

15 [0031] Windows 108 and 1 10 can be associated with a number of window 

attributes. For example, window 110 can include x, y, and z position attributes 
that specify the 3D position of the center of window 110 within 3D display model 
102, as well as a rotation attributes that specify rotations of window 1 10 around 
horizontal and vertical axes. Window 1 10 can also be associated with scaling 

20 factor, translucency and shape attributes. 

[0032] 3D objects within 3D display model 102 are viewed from a 
viewpoint 106 through a 2D display 104, which is represented by a 2D rectangle 
within 3D display model 102. During the rendering process, various well-known 
techniques, such as ray tracing, are used to map objects from 3D display model 

25 102 into corresponding locations in 2D display 104. 

[0033] The bottom portion of FIG. 1 illustrates some of the system 
components that make it possible to map 2D windows into 3D display model 102 
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in accordance with an embodiment of the present invention. Referring to FIG. 1 , 
applications 1 14 and 1 16 are associated with windows 108 and 110, respectively. 
A number of components are involved in facilitating this association. In 
particular, applications 1 14 and 1 16 are associated with xclients 118 and 120, 
5 respectively. Xclients 118 and 120 in turn interact with xserver 122, which 
includes an associated xwindow manager. These components work together to 
render output bitmaps 124 and 126 for applications 1 14 and 1 16 to be displayed in 
windows 108 and 110, respectively. These bitmaps 124 and 126 are maintained 
within back buffer 128. 

10 [0034] Code module 130 causes bitmaps 124 and 126 to be displayed on 

corresponding windows 108 and 110. More specifically, code module 130 
retrieves bitmap 126 and coverts it into a texture 132, which is displayed on the 
front face of window 110. This is accomplished though interactions with 3D 
scene manager 134. Bitmap 124 is similarly mapped into window 108. 

1 5 [0035] 3D scene manager 134 can also received input from a 2D pointing 

device, such as mouse 136, and can communicate this input to applications 1 14 
and 1 16 in the following way. 3D scene manger 134 first receives an input 
specifying a 2D offset from mouse 136 (step 202). Next, the system uses this 2D 
offset to move a cursor 109 to a new position (x\,y\) in 2D display 104 (step 204). 

20 [0036] The system then determines if cursor 109 overlaps a window in 3D 

display model 102 (step 206). This can be accomplished by projecting a ray 107 
from viewpoint 106 through cursor 109 and then determining if the ray intersects 
a window. If there is no overlap, the process is complete. 

[0037] Otherwise, if there is overlap, the system uses the 3D position 

25 {x^yi^i) within display model 102 where ray 107 intersects window 1 10, as well 
as attributes of window 1 10, such as position and rotation attributes, to determine 
the 2D position {x^y{) of this intersection with respect to a 2D coordinate system 
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of window 110 (step 208). The system then communicates this 2D position {x^yi) 
to application 1 16, which is associated with window 110 (step 210). 

[0038] 3D scene manger 134 is also coupled to bump map 135. Bump 
map 135 allows 3D scene manager 134 to apply a bump map with 3D texture 
5 effects to the surface of objects within 3D display model 102, such as 3D object 
111. 

[0039] Various user inputs, for example through mouse 136 or a keyboard, 

can be used to manipulate windows within 3D display model 102. 

i 

10 Using Visual Effects to Convey Usage Context 

[0040] FIG. 3 presents a flow chart illustrating the process of using visual 
effects to convey usage context of computational resources in accordance with an 
embodiment of the present invention. The system starts by determining the usage 
context of a computational resource (step 302). Note that the computational 

1 5 resource can include any type of entity within (or associated with) a centralized or 
distributed computer system. For example, a computational resource can include 
a file, a folder, or an application. Also note that the usage context of the 
computational resource can specify any attribute of a computational resource, such 
as a time at which the computational resource was created, a last time the 

20 computational resource was accessed, a frequency of usage of the computational 
resource, an owner of the computational resource, a location of the computational 
resource within the 3D display environment, and a group membership of the 
computational resource. 

[0041] Next, the system determines a visual effects corresponding to the 

25 usage context to apply to the 3D object which represents the computational 

resource (step 304). Note that these visual effects can include any type of visual 
effect that can be created within a 3D display environment, such as a location of 
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the 3D object, an orientation of the 3D object, a color of the 3D object, a texture 
of the 3D object, a surface pattern on the 3D object, a bump map for the 3D 
object, a size of the 3D object, lighting for the 3D object, transparency of the 3D 
object, reflectiveness of the 3D object, and simulated degradation of the 3D 
5 object. For example, simulated degradation of the object can indicate that a large 
amount of time has passed since the corresponding computational resource was 
last accessed. Furthermore, a specific bump map can be applied to the 3D object, 
along with color to indicate wear on an object that is accessed frequently. Finally, 
the system applies the visual effects to the 3D object within the 3D workspace 
10 (step 306). 

[0042] Note that in addition to the visual effects that are generated based 
on usage context, the system may also apply visual effects based on actions taken 
by a user. For example, a user may make markings on the surface of objects or 
fold the corners of objects for specific purposes. These markings or folds may be 
15 incorporated along with the other visual effects being applied to the object. 
Furthermore, these markings or folds may themselves be subject to continuing 
visual effects such as aging. 



Using Visual Effects to Indicate Groupings of Objects 

20 [0043] FIG. 4A illustrates a group of windows in 3D display model 102 in 

accordance with an embodiment of the present invention. 3D display model 102 
contains windows 401-404, which are arranged in a typical 2D application layout, 
which does not convey any usage context with regards to group membership. 

[0044] In contrast, FIG. 4B illustrates a group of windows in 3D display 

25 model 102 with visual effects indicating groupings of windows in accordance 
with an embodiment of the present invention. Windows 401 and 402 belong to 
the same application or resource, or are related in some manner, and are oriented 
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to the same viewing angle. Likewise, windows 403 and 404 belong to the same 
application or resource, or are related in some manner, and are oriented to the 
same viewing angle. However, windows 401 and 402 are not related to windows 
403 and 404, so the orientation of windows 403 and 404 is different than that of 
5 windows 401 and 402. Note that other visual effects mentioned previously may 
also be used to indicate group membership. For example, windows 403 and 404 
may be highlighted with a scoped spotlight to indicate their group membership. 

[0045] FIG. 4C presents a flow chart illustrating the process of using 
visual effects to convey usage context of a group of computational resources in 

10 accordance with an embodiment of the present invention. The system starts by 
analyzing the usage context of windows 401-404 (step 406). Next, the system 
determines relationships between the windows in 3D-display model 102 (step 
408). Once the relationships have been determined, the system applies visual 
effects to windows 401 to 404 to indicate group membership (step 410). Note that 

1 5 as the visual effects for one window are altered, the visual effects for all related 
windows may be altered to maintain visual confirmation of the relationship. For 
instance, in one embodiment of the present invention, as a cursor moves over 
window 403, the orientation of window 403 may change so that it directly faces 
viewpoint 106. Because window 403 and window 404 are related, the orientation 

20 of window 404 may also change so that it directly faces viewpoint 106. 

Furthermore, when the cursor is moved off of window 403, windows 403 and 404 
may both revert back to their original orientation. 



Virtual Workspaces 

25 [0046] FIG. 5 illustrates a number of virtual workspaces within 3D display 

model 102 in accordance with an embodiment of the present invention. Note that 
3D display model 102 contains many different 3D objects, many of which are not 
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visible from viewpoint 106 at any given time. Objects that share some context 
and are related may be moved close together within 3D model 102 to create 
virtual workspaces. For example, in 3D display model 102, some objects are 
arranged to create workspace 502 and other objects are arranged to create 
5 workspace 504. Viewpoint 106 may be turned and/or moved through 3D display 
model to shift the focus from one workspace to another. Note that objects may 
also be part of multiple workspaces and may also move from workspace to 
workspace while viewpoint 106 is being moved. 



10 Environment Mapping 

[0047] FIG. 6 presents a flow chart illustrating environment mapping in 
accordance with an embodiment of the present invention. The system starts by 
receiving a command to apply visual effects to a 3D object (step 602). Note that 
any of the visual effects previously mentioned may be applied. In addition, the 

1 5 system analyzes the environment around the 3D object (step 604). For example, 
this can include checking for nearby objects, checking the time of day, and 
checking the background color. Once the surrounding environment has been 
analyzed, the system applies the visual effects to the 3D object along with visual 
effects that reflect the immediate environment around the 3D object (step 606). 

20 For example, if another object was close to the 3D object, the reflection of the 
other object might be visible on the surface of the 3D object. Environmental 
mapping allows the 3D display model to more closely model the real world 
including time of day, lighting, point of view, and location. 

[0048] The foregoing descriptions of embodiments of the present 

25 invention have been presented for purposes of illustration and description only. 
They are not intended to be exhaustive or to limit the present invention to the 
forms disclosed. Accordingly, many modifications and variations will be apparent 
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to practitioners skilled in the art. Additionally, the above disclosure is not 
intended to limit the present invention. The scope of the present invention is 
defined by the appended claims. 
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